fix(wasm): size the query buffer from the bytes actually written; reject embedded NUL - #173

Summary

Fixes #169. Before this change, the wrapper could parse less than the query it was given and report no error.

stringToPtr sized the input buffer with Emscripten's lengthBytesUTF8 and then wrote it with stringToUTF8. The two count unpaired surrogates differently:

• lengthBytesUTF8 counts 4 bytes for every two code units.
• stringToUTF8 writes 3 bytes for each code unit.

So the buffer came out too small, the write stopped early, and the end of the query was dropped. Depending on where the cut fell, the second statement disappeared or the parser threw a bogus syntax error. An embedded U+0000 ended the C string early in the same way.

// before
const len = wasmModule.lengthBytesUTF8(str) + 1;   // undercounts lone surrogates
wasmModule.stringToUTF8(str, ptr, len);            // stops at len → tail dropped
// after
if (str.includes('\0')) throw new Error('Query must not contain a NUL (U+0000) character');
const wellFormed = str.replace(/[\uD800-\uDBFF][\uDC00-\uDFFF]|[\uD800-\uDFFF]/g,
                               (s) => (s.length === 2 ? s : '\uFFFD'));  // pairs kept
const size = wasmModule.lengthBytesUTF8(wellFormed) + 1;  // agrees with stringToUTF8 now
wasmModule.stringToUTF8(wellFormed, ptr, size);

The change is made in templates/index.ts and templates/full/index.ts, then copied into versions/*/src/index.ts. No WASM, C, Makefile, proto or type changes.

Why not TextEncoder: an earlier revision encoded the input with TextEncoder. Jest's jsdom environment has no global TextEncoder, so require('libpg-query') threw ReferenceError there. Published 18.1.5 works in jsdom. The regex plus Emscripten's own helpers needs no runtime global. On 1M random strings it gives the same bytes as TextEncoder.

Why replace unpaired surrogates instead of throwing: Node Postgres drivers encode strings with the WHATWG UTF-8 encoder, which turns an unpaired surrogate into U+FFFD. Postgres therefore receives U+FFFD, and the parse tree now reflects exactly those bytes. UTF-16 offsets don't move, because one code unit becomes one code unit.

Why throw on NUL: Postgres can't receive one in query text. Against a real PG 14 server, pg gets 08P01 invalid message format for a NUL in query text, on both the simple and extended protocol.

Evidence

test/input-encoding.test.js was added to every versions/* package and wired into each test script.

• New tests against published packages: published libpg-query@18.1.5 and 13.7.3 (same WASM, old wrapper) pass 3 of 11; the fixed wrapper passes 11/11. The new no-TextEncoder test also fails on this PR's earlier TextEncoder revision.
• Same output elsewhere: the old and new wrappers ran around the same WASM over every .sql file in pgsql-parser (557 files, about 110k inputs per version). That is 662k comparisons, about 40k of them error results, with 0 differences:
  • v18: parseSync, fingerprintSync, normalizeSync and scanSync;
  • v17 and v13: parseSync.
• Speed: no measurable difference on that corpus (within ±3% run-to-run noise).
• Downstream: pgsql-parser's full suite (3,349 tests) gives identical results with this wrapper and with published 18.1.5. Its one failure, a plpgsql-parse snapshot, fails the same way on both.
• jsdom: under jest 29 + jest-environment-jsdom, the fixed wrapper loads and parses.
• CI: every version on Linux, macOS and Windows.

Link to Devin session: https://app.devin.ai/sessions/bcd8b2f791164247acb7a659b481efc2
Open in Devin Desktop: https://app.devin.ai/desktop/session/bcd8b2f791164247acb7a659b481efc2?variant=devin
Requested by: @pyramation